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Abstract — We consider the problem of source coding with 
receiver side information for the simple network proposed by 
R. Gray and A. Wyner in 1974. In this network, a transmitter 
must reliably transport the output of two correlated information 
sources to two receivers using three noiseless channels: a public 
channel which connects the transmitter to both receivers, and 
two private channels which connect the transmitter directly to 
each receiver. We extend Gray and Wyner's original problem by 
permitting side information to be present at each receiver. We 
derive inner and outer bounds for the achievable rate region and, 
for three special cases, we show that the outer bound is tight. 

L Introduction 

The field of network source coding is centered on the 
following problem: given a noiseless communications network 
and a set of information sources, what is the best way to 
compress the output of each source for efficient and reliable 
transportation over the network? A solution to this type of 
problem needs to remove any temporal redundancy in each 
source, exploit any statistical correlations between different 
sources and optimize the use of limited channel capacities. 

In network source coding, a code is a collection of rules that 
define how the output of each source is to be compressed, 
transported over the network and reconstructed. A code is 
said to be reliable if the output of each source can be 
reconstructed without error at each of its intended destinations. 
The performance of a reliable code is measured by the rates 
at which it sends data over each channel; an optimal code will 
send data at the smallest rates and thereby consume the least 
network capacity. An ordered collection of rates (one for each 
channel) is said to be achievable if there exists a reliable code 
which operates at these rates. The set of all achievable rates 
3i is called the achievable rate region of the network, and its 
lower boundary ^ provides a performance benchmark for the 
comparison of reliable codes. 

NICTA is funded by the Australian Government as represented by the 
Department of Broadband, Communications and the Digital Economy and the 
Australian Reseai'ch Council through the ICT Centre of Excellence program. 

The work presented in this paper was undertaken by R. Timo while on 
visit at the Institute for Telecommunications Research, the University of South 
Australia, and Bell Laboratories, Alcatel-Lucent. R. Timo's travel was funded 
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The achievable rate region 3^ is known for a small ad-hoc 
collection of networks; for most "real world" networks, M 
is unknown [1]. With the exception of [2], achievable rate 
regions have been studied on a network-by-network basis; 
researchers have designed and studied simple networks which 
isolate particular problems of interest. Two notable examples 
are: the separate coding of correlated sources [3], and the 
sharing of a finite capacity channel between multiple users [4]. 
It is hoped that solutions to these simple networks will yield 
practical and efficient codes for larger networks. 
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Fig. 1. Figure shows the network source coding problem proposed by R. 
Gray and A. Wyner [5]. The transmitter is connected to two receivers via 
three noiseless channels. The sequences {Xi\ and {Yi} are to be encoded 
at the transmitter, transported over the network and decoded at the x and 
j/-receivers respectively. In this paper, we study an extension of this problem 
where "side information" {Ui} and {Vi} are present at each receiver. These 
additional information sources are marked with dashed lines in the figure. 

We study the achievable rate region ^ of the network 
shown in Figure [T] A transmitter must transport the output of 
two correlated sources to two receivers using three noiseless 
channels: a public channel which connects the transmitter to 
both receivers, and two private channels which connect the 
transmitter directly to each receiver. The achievable rate region 
^ of this network was found by R. Gray and A. Wyner [5] 
in 1974. They showed that an optimal code should endeavor 
to use the public channel to transport information common to 
both sources. As we will see, the intuition of this solution is 
lost when side information is introduced at each receiver; in 
particular, it is not clear how one should decompose the output 
of each source for transmission over the three channels. 



An outline of the paper is as follows. To fix ideas, we 
briefly review [5] in Section [III In Section |III1 we formally 
define M for the network with side information. In Sections HVl 
and |Vl we derive outer and inner bounds for Si, respectively. 
In Section IVII we ascertain M for one source, a degraded 
network and a complementary delivery network respectively. 
Finally, we conclude the paper in Section IVIII 

II. The Gray-Wyner Problem 

Consider the network (without receiver side information) 
shown in Figure [T] We denote the capacities (in bits per 
second) of channels 0, 1 and 2 by Co, C\ and C2 respectively. 
Finally, let and ^ be finite alphabets, and let JT" and ^" 
denote their respective n-fold cartesian product spaces. 

Suppose {(X„K,)} = % = 1,2,...} is a 

sequence of independent and identically distributed (i.i.d.) 

x valued random variables emitted by a discrete mem- 
oryless source Qxy(a;,?;) = ProbfX = x,Y = y\. Suppose 
further that the random sequence { {Xi , ) } appears at the 
transmitter at the rate of one per second. It is desired that the 
transmitter delivers a reliable reproduction {Xi} = {Xi, i = 
1, 2, . . .} of the sequence {Xi} to the x-receiver, and a reliable 
reproduction {Yi} = {K^; i — 1,2,...} of the sequence 
{Yi} to the j/-receiver Assuming no delay constraints and 
unlimited computational power at the transmitter and receivers, 
the main problem is to ascertain which channel capacity 
triples (Co, Ci, C2) are both necessary and sufficient for each 
sequence to be reliably transported to its intended destination. 

We assume the classic n-block source coding model where 
the sequence {{Xi,Yi)} is parsed and transported over the 
network in message blocks of length n (for some large integer 
n). Let (X", y") = {Xi,Yi), ^2)^. • ■ , (^n, Jn) denote 
the message at the transmitter, and let X" — Xi,X2, ■ ■ ■ , Xn 
and = Yi,Y2, . . . ,Yn denote the reconstructed messages 
at the X and y-receivers respectively. 

For each i = 0, 1, 2, let — {1, 2, . . . , |./#i|} be a finite 
index set for use on channel i. A network source code is a 
collection of mappings (e*-"-*, dy"''), where e^"^ : x 
"3^" ^0 X X -^2 is the encoder at the transmitter; 
di"'' : ^0 X ^1 is the decoder at the x-receiver; 

and dy"^ : x ^2 "3^" is the decoder at the y-receiver 
The transmitter encodes the pair with three indices 

(A/o, Ml, A/2) = e(")(X", F") which are sent over channels 
0, 1 and 2 respectively. After receiving indices Mq and Mi, 
the x-receiver reconstructs X" — di"-' (A/o, A/i). Similarly, 
after receiving indices Mq and A/2, the y-receiver reconstructs 
F" = S^\Mq, M2). An error is said to occur if either X" ^ 
X" or F" 7^ F", and the code is said to operate at a rate 
of (l/n)log2 bits per source symbol on channel i (for 
z = 0,l,2). 

A triple of rates {Rq, Ri, R2) is said to be achievable 
if there exists a sequence of codes {(e^"-',(ii"\(iy"''); n — 
1,2,...} such that the probability of error approaches zero 
and (l/n) log \ approaches Ri (for i = 0, 1, 2) as n goes 
to infinity. 



Let denote the set of all achievable rate triples. It can 

be shown that Sicw is a closed convex subset of Euclidean 
three space, which is completely defined by its lower boundary 
^Gw [5]: 

S^GW — {{Ro, Rl, R2) £ ^GW ■ {Ro/Ri, R2) g S?gw, 
R, <R^{^^ 0, 1, 2)^R,= R, (^ = 0, 1, 2)} . 

Given Qxy and a network with capacity triple (Co,Ci, 
C2), the sequences {Xi} and {F} may be reliably recon- 
structed at the X and y-receivers respectively if and only if 
(Co,Ci,C2) Hes above ^gw'-> thus, S^gw defines exactly 
those capacity triples which are both necessary and sufficient 
for reliable communication. 

Gray and Wyner [5] showed that to achieve rates 
{Rq, Ri, R2) which lie on the lower boundary S?gw, the ca- 
pacity of channel should be prioritized for use by information 
common to both {Xi} and {F}. Specifically, they designed 
a coding scheme which used an auxiliary random variable W 
to represent the information transported over channel 0, and 
they showed any {Rq, Ri, R2) € ^gw may be achieved by 
optimizing over the choice of W. 

The formal description of S^gw in terms of W is as follow. 
Let ^ be a finite alphabet of cardinality |#'|<|jr||^| + 2, 
and let ^gw denote the family of probability functions on 
W X ^ X such that Y.wPi'^^^^v) = Qxvix^y). Now, 
for each p G 3^gw, let 

r i?o > Ip{X,Y;W) \ 

^GW^ U^o,i?i,i?2) : Ri > Hp{X\W) \ , 
[ R2 > Hp{Y\W) J 

where /p(-; •) denotes mutual information and Hp{-\-) denotes 
conditional entropy (with respect to p). 

Lemma 1: [5, Thm. 4] The achievable rate region Mgw 
of the Gray-Wyner Network is given by 

^GW=[ U ^'^^l ) , 
\p£.9'gw I 

where {-Y denotes the set closure operation. 

It follows from Lemma[T]that Mgw is completely described 
by a single coding scheme which makes use of an auxiliary 
random variable W . As we will see, this coding scheme 
extends, in a natural way, to the network with side information. 
Unfortunately, however, this extension does not appear to 
completely describe the corresponding rate region. 

III. Extension to the Side Information Case 

Suppose ^, ^, and r are finite sets, and let JT", 
and 1^" denote their respective n-fold cartesian product 
spaces. Suppose further that {(X^, F^, V^)} is a sequence 
of i.i.d. X xW x'^i/ x'V valued random variables emitted by a 
discrete memoryless source QxYC/y (2;, y, u, w) = Prob[X = 
x,Y — y,U — u,V — u] . Finally, for each i ~ 0,1,2, let 
= {1,2,..., \^i\} be a finite index set for channel i. 
As before, a source code is a collection of mappings 
(e(") , di"^ , 4"^ ), where e^") : ^" x ^" xJ^^xJ^2 is 



the encoder at the transmitter; : Ji^ x Jl\ x -h. J:'" is 
the decoder at the x-receiver; and d^"^ : x -^2 x 
is the decoder at the y-receiver. The transmitter encodes the 
pair with indices {Mq,Mi,M2) = e(")(X",y") 

which are sent over channels 0, 1 and 2 respectively. After 
receiving indices Mq and Mi as-well-as side information 
[/", the a;-receiver reconstructs X" = di"^ (Mq, Mi, [/"). 
Similarly, after receiving Mq, M2 and V", the y-receiver 
reconstructs = 4"H^o, M2, V"). 

An error occurs if either X" ^ X" or ^ F". Let 
Pg ^, A Prob[X" ^ X"], Pe,y = Prob[y" ^ Y"] and P, = 

maxIPej,, -Pe,j/}- 

Definition 1 (Achievable Rate): A rate triple {Rq, Ri, R2) 
is said to be achievable if, for arbitrary e > and suf- 
ficiently large n, there exists a code (e^"-*, rf^,"'') with 
parameters {n,\^o\,\^i\,\^2\, Pe) such that Pe < e and 
(1/n) log < Rt + e for alH = 0, 1, 2. We let ^ denote 
the set of all achievable rate triples. 

IV. An Outer Bound 

Suppose is a finite set of cardinaHty |yr| < | jr| + 3 
and ^ is the family of probability functions on W x ^ x 

x'^ x'y such that x, y, u, = ^(u'la;, y)p{x, y, u, v) 
and 

QxYuv{x,y,u,v) = ^ p{w,x,y,u,v) 
for all p G Now, for each p ^ ^ let 

i?o > max{/p(X,y;W^|t/),/p(X,y;W^|F)} ] 

i?o + i?i > ma^{lp{X,Y;W\U),IpiX,Y;W\V)} 

+HpiX\W,U), 
R0 + R2 > max {lp{X,Y;W\U),Ip{X,Y;W\V)} 

+HpiY\W,V). 

Theorem! (Outer Bound): If {Rq,Ri,R2) is an achiev- 
able rate triple, then there exists a p e ^ such that 



{Ro, Ri, R2) e ' 



A. Proof Outline: Theorem Q] 

We show: if {(e*-"-*, c?i"\ is a sequence of codes 

where Pe ^ as n ^ 00, then there exists ap G £P such that 

((1/71) log Kol, (l/n)log|^i|, {l/n)\og\^2\) 

Suppose (e("\dl"\4"^) is a code with (Mo,Mi,M2) = 
e("'(X",r"), X" = di"^(Mo,Mi,C/") and f" = 
4"^(Mo,M2,y"), then 

logl^ol > -ff(Mo|C/") > /(X",y";Mo|f/") 

n 

= ^/(x„y,;Afo,xr\^r\c/r\c/r+i|[/o (d 

z^l 

n n 

> ^ y,; Mo\U,) = I{Xr,Y,- , (2) 



where ([T]) follows because {{Xi,Yi,Ui,Vi)} is drawn in an 
i.i.d. fashion and follows by setting Wt = Mq. Similarly, 

n 

log Kol > li^^^Y,; . (3) 

i=l 

On applying Fano's Inequality [6, Pg. 37] we get 

iJ(X"|Afo, Ml, [/") < < n^(Pe, n) , (4) 

where 5(Pe,n) = (1/n) + Pe log IS"!!^]. Similarly, we also 
have that H(Y"\Mq, Mi,V") < nS{Pe,n). 

Now consider the series of Shannon (in)equaUties ^ 
through Note, Q follows because {{Xi,Yi,Ui,V)} 

is drawn in an i.i.d. fashion and (fTTT i follows since Mq & 
{Xi, Yi) {Ui, Vi) forms a Markov Chain and dUi. From (fTOt 
and ( fT2] i. it respectively follows that 

^(^ log 1^0 1 + log 1 j 



i— 1 



and 



- flog 1^0 1 + log M 
1 " 

-y^\l{X,,Y,-W,\V,) + H{X,\W,,U,) -5{P,,n) . 



> 



Note, (1/n) [log \ ^q \ + log |.'#2|] may be bound in a similar 
manner Following the time sharing principle given in [5, 
Pg. 1709], we may now construct a p e ^ such that each 
inequality in the theorem holds as n 00 and Pe 
0. Finally, we may bound the cardinality of the auxiliary 
random variable W using the support lemma of Ahlswede 
and Korner [7, Lemma 3]. 

V. An Inner Bound 

A natural extension of the code proposed by Gray and 
Wyner [5] yields the following inner bound for 

Let W and ^ be defined as in Section [III] For p e ^, let 



|(Po, Ri,R2) ■ 



Ro > 
Ri > 
R2 > 



max {ipiX, Y; W\U), Ip{X, Y; W\V)] 

Hp{X\W,U), 

Hp{Y\W,V). 



and ^„ = ( Upg^ 

Theorem 2: M D I^in- 
Remark 1: If i7 = V", then ^ = 

Remark 2: Suppose {X, Y)^U-e-V forms a Markov chain. 
It can be shown that a sum rate Rq + Ri+ R2 is achievable if 
and only if Po + Pi+i?2 > H(Y\V) + H{X\Y,U). (See [8] 
for the special case where V = constant.) We may setW = Y 
in Theorem |2] to achieve this sum rate. 

Remark 3: Suppose X = Y. Sgarro [9] showed that the 
sum rate Pq + Pi + P2 is achievable if and only if Pq + Pi + 



log Kol + log Kil > i/(Mo, Ml) = H{Mo, M,\U") + /(Mq, Mi; {/") 
> /(X",y";Mo,Mi|C/") +/(Afo,Mi;;7") 



= ^ r,; Mo, Ml, Xj-i, r^i, Ur\U^+,\U,) + /([/,; Mo, Mi, C/^i)] 

71 

> Y,; Mo, Ml, Ul-\Ul!^,\Ui) + /(C/.; Mq)] 

n 

= [l{X,,Yf,Mo\U,) + I{X,,Y,- Mi,Ur\ U^+,\Mo, Ui) + /(C/,; Mq)] 

i=l 
n 

> Y [I{X,,Y,:M^\U,) + I{X,;Mi, Ul^U^^^M, U,) + I{U,- Mo)] 

n 

= Y [l{X,,Y,-M^,\Vi) + H{X,\Mo, U,) ~ n6{Pe,n)\ 

i=l 

n 

= Y [l{X„Yf,W,\V,) + H{X,\W^,U^)-n6{Pe,n)] 



(5) 
(6) 

(7) 
(8) 
(9) 
(10) 

(11) 
(12) 



i?2 > max{H{X\U), H{X\V)}. We may set W 
in Theorem 12] to achieve this sum rate. 

Remark 4: Suppose U ~Y and V ^ X. Wyner et. al. [4] 
showed that the sum rate i?o + i?i + i?2 is achievable if and 
only if i?o + ^1 + -R2 > max{ir(X|r), i7(r|X)}. We may 
set W — {X,Y) in Theorem |2] to achieve this sum rate. 

Remark 5: The code, which yields the achievability of S^in, 
is essentially a version of Heegard and Berger's "triple rate 
split code" given in [10, Thm. 2]. Indeed, we note that the 
problem of minimizing the sum rate i?o + ^1 + R2 is a special 
case of the two receiver generalized Kaspi-Heegard-Berger 
problem [10, Sec. VII]. 

A. Proof Outline: Theorem |2] 

1) Code Construction: Suppose p £ Let R'^, R[ and 
i?2 be non-negative integers whose values will be chosen 
later. Generate 2"^o independent lu-codewords of length n 
by choosing symbols i.i.d. from W according to (the W- 
marginal of p). Label the resulting code book with the index 
m'o: = {^"(771;,) ■.l<m'o< 2"^o}. Similarly, generate 
2"^i and 2"^^ independent x and y-codewords using px and 
Py respectively: '^x = {x'"'{m[) : 1 < m[ < 2"^i}, and 

^ {y" (my : 1 < TO^ < 2"-«2}. 
Uniformly at random assign to each G "^^V a "bin label" 
from the set .^0 = {1,2,... , 2L"«oJ }, and let /i^ : ^ 
./#o denote the induced mapping. Let ^-^{nio) denote the set 
of w-codewords with bin label mo: ^^(mo) = {w" G '^yy : 
h'^{w'^) = mo}, and let ^S-^ denote the collection of all w- 
bins. In the same way, assign one of 2L"^iJ and 2L"^2j ^jjj 
labels to each x and y-codeword, and define /i^, h^, 
and ^13/. 

2) Encoding: The encoder assumes the messages x", y", 
w" and v" emitted by the source are e-strong joint typical; 
that is, (x",j/",w",?;") G A*e'''''\pxYUv)- Let Ei denote the 



X — Y event where this assumption is false. Then [6, Lem. 10.6.1] 



Pr:[Ei] < ei{n, JT x ^ x ^ x r) , 



(13) 



where ei{n, ^ x ^ x ^ x r) ^ in n for fixed e > 0. 

The transmitter looks for a w"(too) G '■^•^ which is e-strong 
joint typical with If two-or-more such codewords 

exist, the transmitter selects the codeword with the smallest 
index. If no such codeword exists, an error is declared and the 
transmitter arbitrarily selects some ^"(mo) G Let E2 
denote this error event. Then [6, Lem. 10.6.2], 

Pr[E2] < e I' ^ ) , (14) 

where 62 ^ as e ^ and n — *■ 00. We assume R'q > 
I{X, Y; W) + 62, so that Pr[£'2] ^ as e and n 00. 
After the transmitter selects w'^irn'o) G it sends the index 
Too — hyy{w'^{m'Q)) on channel 0. 

The transmitter looks for a x"{m[) G '^x such that 
x"{m'i) — x". If two-or-more such codewords exist, the 
transmitter selects the codeword with the smallest index. If no 
such codeword exists, an error is declared and the transmitter 
arbitrarily selects some ^"(to'i) G Let E^ j; denote this 
error event. Then, 



Pi- IB: 



3,'x 



< e 



-(2"«i 



i2-"<"(^) + ^3,x) 



) 



(15) 



where £3 2; ^ as e ^ and n ^ 00. Choose Rq > H{X) + 
£3 arbitrarily, so that Pr[£^3 ^ as e ^ and n 
CO. The transmitter encode y" is a similar fashion, and sends 
TOi ~ h X [x'"' {m'^)) and 7712 — {y^'^ (7112)) on channels 1 
and 2 respectively. 

3) Decoding: Given ttjo and m", the X-receiver looks for 
a unique u;" G ^^(ttio) which is jointly typical with u". If 
no such codeword can be found, an error is declared and the 
decoder arbitrarily selects some ui" G £§-g/{mo). Let 



• £'4.2;: the codeword w"'{mQ) chosen by the transmitter is 
not jointly typical with w", and 

• E^^x- there are two-or-more w-codewords in S^-g/imo) 
which are jointly typical with u". 

Consider E^^^. Since W {X,Y) e U forms a Markov 
Chain under p, we have that [6, Lem. 15.8.1] 



(16) 



where £4^x ^ as n — > 00. 

Now consider E^^^. We have that u" e A*e''"'\Pu)- As 
before, the probability that a randomly generated w-codeword 
is jointly typical with u" is upper bound by 2^"'^-^'^^''^'+'^^'^\ 
where es ^. ^ as n — > cxo. Moreover, the number of 
codewords in each bin is at most 2"(^o^^o) + e^i j., where 
£5', a: ^ as n ^ 00 [11, Pg. 2766]. Hence, 

Pr[E5,x] < 2-"(«o-«o+^(M';C/)-^5..) +^5,^^ 

Wensed Ro-R'o + I{W;U)-€5,x > 0, so that Pr[i;5 J ^ 
as n ^ 00. This requires 

Ro>R'o- I{W; U) + £5,. 

>I{X,Y-W)-I{W-U)+e2 + e5^x (17) 
= I{X,Y,U-W)-I{W;U) + e2 + ez,x (18) 

= I{X,Y;W\U)+€2+€^,x , (19) 

where (fTTT i follows because we selected i?Q > I{X, Y\ W) + 
£2, ( fTST i follows because €^ (X, F) «- L/ forms a Markov 
Chain, and ( fT9] l follows from the chain rule for mutual 
information. Similarly, the y-receiver will correctly find a w- 
codeword with high probability if Rq > I{X, Y;W\V) + e2 + 
e^^y, where £5^5^ ^ as £ ^ and n — > 0. 

Given w", mi and it", the X-receiver looks for a unique 
£ ^ac{mi) which is jointly typical with and u''\ If 
there exists two-or-more such codewords, an error is declared 
and the decoder arbitrarily selects some x" £ ,^^(mi). Let 
Eq^x denote this error event. It follows that 



(20) 



where sq^x — > and £6',^ — ^ as £ ^ and n ^ 00. If _Ri > 
H{X\w',U) + ee,x it follows from ^ that Pr[£'6,:r] ^ as 
n 00. Similarly, the y-receiver will correctly find with 
high probability if R2 > H{Y\W, V) + ee^y, where eg^y 
as £ ^ and n ^ 0. 

VI. Three Simple Networks 
A. Two Descriptions of M when X — Y 

Let ^ and ^^^^j be defined as in Section HV] 
Theorem 3: If X = Y, then ^ = {^^^,^^^^^2^" . 
Now suppose and S§ are finite sets of cardinalities |^| < 
jr| + 1 and 1^1 < I J"| + 1. Let ^* denote the family of 
probability functions onjz/x^xJTx^xy such that 
p{a, b, X, u, v) = p{a, b\x)p{x, u, v) and 

Qxuvix,u,v)= ^ p{a,b,x,u,v) . 



For each p G let 

^(^')* = {(i?o,i?i,i?2): 

i?o > m'Ayi{Hp{X\A,U),Hp{X\B,V)} 
Ri > Ip{X;A\U), 
R2 > Ip{X;B\V). 

Theorem 4: If X = Y, then ^ = {Upe^^-^^P^*)" ■ 

B. ^ for a Type of Degraded Network 

Let ^ and ^^^^^ be defined as in Section |IV] 
Theorem 5: If Y = {X, Z) and {X,Z) <^U <^V forms a 
Markov Chain, then ^ = (Upe^^^^i 



C. ^% for a Complementary Delivery Network 

Let ^ and ^^^^^ be defined as in Section |IV] 

Theorem 6: \f U = Y wA V = X, then ^ = 

Now suppose jz/ and ^ are finite sets of cardinalities |^| < 
JT'll^l + 1 and \^\ < \SC\\W\ + 1. Let ^** denote the set 
of probability functions ox\ si x^xJTx^^ such that 

Q.XY{x,y)= ^ p{a,b,x,y) 
is true for all (a;, y) and p e ^**. For each p G ^**, let 



^(P)** = {(i?0,i?i,i?2) 



i?o > max{i/p(X|A,r),i/p(y|S,X)} 
Ri > Ip{X;A\Y), 
R2 > Ip{Y;B\X). 



Theorem 7: If U 

{Up^^,,M^p>*y. 



Y and V 



X, then 



VII. Conclusion 

We investigated the achievable rate region ^ of a simple 
network with side information present at each receiver. Our 
first theorem gave an outer bound which, for three simple 
networks, was shown to be equal to Our second result 
gave an inner bound which was obtained via an extension of 
the coding theorem given by Gray and Wyner [5]. 
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